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ABSTRACT 

Research  in  command  and  control  is  advancing  rapidly  through  a  campaign  of  laboratory 
experimentation  using  the  ELICIT  (Experimental  Laboratory  for  Investigating  Collaboration, 
Information-sharing,  and  Trust)  multiplayer  online  counterterrorism  intelligence  game.  In  most 
ELICIT  experiments,  participants  play  the  game  through  a  Web  interface  and  interact  with  one 
another  solely  through  textual  information  exchange.  This  mirrors  in  large  part  the  network¬ 
centric  environment  associated  with  most  counterterrorism  intelligence  work  in  practice. 
However,  we  argue  that  a  more  immersive  virtual  environment  offers  potential  to  improve 
performance.  The  research  described  in  this  article  addresses  the  preliminary  design  and 
prototyping  of  an  immersive  and  dynamic,  virtual  environment  for  ELICIT  experimentation,  in 
which  participants  play  the  game  through  avatars  representing  them.  We  summarize  a  number  of 
alternate  virtual  enviromnent  platforms  available  to  researchers  for  work  such  as  this,  and  we 
explain  the  rationale  for  the  specific  environment  selected  for  this  project.  Additionally,  we 
describe  the  prototypical  design  strategies  for  adapting  game  play  into  virtual  environments  and 
characterize  preliminary  case  examples  to  understand  the  differences  and  potential  benefits  of 
adapting  ELICIT  game  play.  This  research  lays  a  foundation  for  moving  experimentation  into 
immersive  virtual  environments. 
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INTRODUCTION 


Modern  military  organizations  have  adapted  and  evolved  over  many  centuries  and  millennia, 
respectively.  Hierarchical  command  and  control  (C2)  organizations  in  particular  have  been 
refined  longitudinally  (e.g.,  through  iterative  combat,  training  and  doctrinal  development)  to 
become  very  reliable  and  effective  at  the  missions  they  were  designed  to  accomplish.  However, 
recent  research  suggests  that  the  Hierarchy  may  not  represent  the  best  organizational  approach  to 
C2  in  all  circumstances  (Nissen,  2005),  particularly  where  the  environment  is  unfamiliar  or 
dynamic.  Indeed,  alternate,  more  flexible  C2  organizational  approaches  such  as  the  Edge  have 
been  proposed  (Alberts  &  Hayes,  2003)  to  overcome  Hierarchy  limitations,  but  the  same  recent 
research  suggests  that  the  Edge  may  not  represent  the  best  organizational  approach  to  C2  in  all 
circumstances  either,  particularly  where  the  environment  is  familiar  and  stable. 

Of  course,  the  Hierarchy  and  Edge  both  represent  organizational  archetypes  (Orr  & 
Nissen,  2006),  each  of  which  offers  considerable  latitude  in  tenns  of  detailed  organizational 
design  and  customization.  For  instance,  recent  research  demonstrates  further  how  the 
performance  of  both  Hierarchy  and  Edge  organizations  is  sensitive  to  factors  such  as  network 
infrastructure,  professional  competency  and  other  factors  that  can  be  affected  through  leadership, 
management  and  investment  (Gateau,  Leweling,  Looney,  &  Nissen,  2007).  With  incessant 
advances  in  information  technology  (IT)  that  appear  to  be  continuing,  one  may  be  able  to 
overcome  the  limitations  inherent  in  Hierarchy,  Edge  or  other  organizations  or  even  enable  such 
organizations  to  adapt — through  IT — to  shifting  conditions. 

This  notion  is  fundamental  to  Network  Centric  Operations  (NCO),  where  people  and 
organizations  operate  principally  in  network-enabled  virtual  environments  as  opposed  to  their 
physical  counterparts.  Unfortunately,  empirical  evidence  to  support  the  asserted  superiority  of 
NCO  remains  sparse,  and  the  capability  enhancing  properties  of  virtual  environments  remain 
more  in  the  domain  of  lore  than  empirical  assessment.  To  remedy  such  empirical  sparseness,  we 
continue  a  campaign  of  laboratory  experimentation  using  the  ELICIT  (Experimental  Laboratory 
for  Investigating  Collaboration,  Information-sharing,  and  Trust)  multiplayer  online 
counterterrorism  intelligence  game  to  understand  the  comparative  advantages  and  disadvantages 
of  alternate  C2  approaches,  organizational  forms,  technologies  and  like  aspects  across  a  range  of 
current  and  anticipated  operational  environments.  Indeed,  drawing  from  substantial  research  in 
modeling  and  virtual  environments,  we  argue  that  a  more  immersive  virtual  environment  offers 
potential  to  improve  performance. 

The  research  described  in  this  article  addresses  the  preliminary  design  and  prototyping  of 
an  immersive  and  dynamic,  virtual  environment  for  ELICIT  experimentation,  in  which 
participants  play  the  game  through  avatars  representing  them.  In  the  balance  of  this  article,  we 
summarize  a  number  of  alternate  virtual  environment  platfonns  available  to  researchers  for  work 
such  as  this,  and  we  explain  the  rationale  for  the  specific  environment  selected  for  this  project. 
Additionally,  we  describe  the  prototypical  design  strategies  for  adapting  game  play  into  virtual 
environments  and  characterize  preliminary  case  examples  to  understand  the  differences  and 
potential  benefits  of  adapting  ELICIT  game  play.  This  research  lays  a  foundation  for  moving 
experimentation  into  immersive  virtual  environments. 
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BACKGROUND 


Virtual  environments  are  dynamic  and  customizable  platforms  for  interaction.  They  present  “a 
synchronous,  persistent  network  of  people,  represented  as  avatars,  facilitated  by  networked 
computers”  (Bell,  2008).  They  allow  users  to  design  and  implement  contextually  relevant  spaces 
for  business,  education,  research,  as  well  as  for  socializing. 

While  akin  to  computer  based  game  environments  and  online  games,  virtual 
environments  allow  for  a  flexibility  of  usage  that  sets  them  apart  from  previous  analogous 
iterations.  The  similarity  between  these  forms  would  lead  one  to  believe  that  virtual  worlds  are  a 
type  of  open-ended  game,  whereas  researchers  suggest  that  online  game  themselves  are  a  fixed 
type  of  virtual  world  (Scroeder,  2008). 

Only  a  short  time  ago,  to  develop  an  immersive  custom  environment  would  have  been 
time  consuming  and  expensive.  As  these  platforms  multiply,  the  use  cost  has  diminished,  and  the 
flexibility  has  increased,  making  this  an  ideal  time  to  pursue  research  requiring  a  virtual 
environment. 

In  considering  the  appropriate  virtual  environment  for  research,  it  is  important  to  ensure 
that  the  selected  platform  is  sustainable  and  not  in  any  danger  of  going  out  of  business.  With  a 
dynamic  new  marketplace  for  “desk-top  virtual  worlds”  (Hudson,  Wood,  Wetsch,  &  Solomon, 
2009),  come  many  different  platforms,  some  of  which  will  inevitably  close.  Virtual  environments 
like  Metaplace,  seen  as  early  competition  to  market  giants  like  Second  Life,  have  recently 
folded.  The  market  for  virtual  world  platforms  continues  to  augment,  change,  with  new  players 
entering  all  the  time.  It  is  therefore  important  to  consider  the  persistence  of  any  platfonn  chosen 
for  this  experimentation. 

While  there  are  dozens  of  virtual  world  platforms  from  which  to  choose  (Robbins,  2010), 
many  of  these  virtual  environments  are  more  akin  to  online  games  in  that  they  are  themed  and 
defined.  Thus  the  list  of  practical  platforms  available  for  fully  customized  environments  is 
greatly  reduced.  The  adaptation  of  ELICIT  into  a  virtual  environment  has  very  specific  project 
parameters  that  again  focus  the  type  of  virtual  world  platform  that  would  be  most  suitable. 

The  ELICIT  game -play  requires  a  very  basic  virtual  environment  set-up,  free  of 
extraneous  elements,  but  also  highly  customizable  in  order  to  adapt  to  various  experimental 
aims.  The  3D  virtual  environment  needs  to  function  in  the  same  way  that  the  real  world  2D 
computer-based  ELICIT  game-play  does,  expect  for  the  addition  of  the  interactive  spatial 
element  to  which  this  research  is  focused.  Players  in  the  virtual  environment  need  to  be  able  to 
receive,  read,  and  evaluate  factoid  clues  that  are  communicated  in  a  text-only  fonnat.  They  need 
to  be  able  to  then  share  what  are  deemed  relevant  factoids  with  other  players,  and  they  need  to  be 
able  to  submit  their  answers  to  the  challenge  when  they  believe  they  have  solved  it. 

The  additional  component  to  the  ELICIT  game-play  experience  in  a  virtual  environment 
is  the  ability  for  participants  to  interact  with  one  another  around  the  game  context.  This  could 
involve  a  range  of  interactivity  from  having  a  common  “bulletin  board”  to  view  games  factoids, 
to  inter-personal  discussions  surrounding  the  factoids,  to  be  determined  by  specific  experimental 
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constraints.  Whatever  the  level  of  interaction  for  each  experiment,  it  is  crucial  that  the  ability  to 
communicate  with  one  another  be  included  in  the  virtual  environment  set-up. 


In  choosing  the  most  suitable  platform  for  adapting  ELICIT,  a  comparison  of  leading  virtual 
worlds  was  conducted  examining  the  key  features  for  each  product  that  are  salient  to  the 
research.  What  is  clear  is  that  at  the  top  strata  of  virtual  platforms  there  is  tremendous  overlap 
between  the  various  providers.  This  research  focuses  on  four  established  virtual  world  platforms 
and  examines  their  offered  features  in  light  of  several  key  experimental  criteria: 

1)  Accessibility:  Virtual  environment  platfonn  must  be  universally  accessible  between 
operating  systems.  The  environment  must  be  intuitive  in  terms  of  its  usage,  reducing  to 
the  bare  minimum  the  training  load  for  all  participants. 

2)  Communication:  Virtual  world  platform  must  allow  for  both  text  and  voice 
communication.  This  communication  apparatus  must  be  straightforward  to  learn  and 
simple  to  use. 

3)  ELICIT  specific  requirements:  ELICIT  game-play  is  a  proctored  clue-based  game  that 
collects  infonnation  about  game-play  in  a  database  so  that  researchers  may  examine 
statistical  records  of  each  game  round.  Currently  the  game  functions  are  handled  by 
JavaScript,  so  ideally,  the  virtual  environment  will  support  the  inclusion  of  JavaScript 
without  significant  re-coding  of  the  environment. 

4)  Customization:  The  virtual  environment  must  support  complete  customization  of  spatial 
aspects.  This  customization  can  preferably  be  done  in  an  expedient  manner. 

Out  of  the  dozens  of  virtual  environment  platforms  in  existence  today,  a  short-list  of  suitable 
products  has  been  detennined  by  examining  the  range  of  platforms  in  lieu  of  the  preceding 
criteria.  While  these  platforms  contain  most  of  the  required  features,  some  are  stronger  than 
others  in  certain  aspects.  For  the  preliminary  research  understanding  the  effects  of  the 
environment  on  ELICIT  game-play,  all  of  the  following  platforms  have  the  core  requirements  to 
proceed  with  the  research: 

Second  Life/  OpenSim 

A  popular  leader  in  the  desktop  virtual  worlds  revolution,  for  many  Second  Life  has  become 
synonymous  with  virtual  environments.  With  built-in  building  tools,  Second  life  offers  a  high 
degree  of  design  flexibility,  which  enables  designers  to  rapidly  prototype  environments  and 
adapt  them  quickly  within  the  world.  OpenSim  is  an  open-source  community  built  around  upon 
the  Second  Life  platform,  while  maintaining  the  look,  feel,  and  functionality  of  the  parent 
platform.  The  basic  communication  requirements  for  the  ELICIT  experiments  are  included  with 
the  Second  Life  platforms.  However,  until  recently,  one  potential  obstacle  to  full  virtual 
integration  of  ELICIT  was  the  lack  of  JavaScript  functionality  natively  present  in  Second  Life 
and  the  Opensim  platfonn.  With  the  recent  release  of  Second  Life  Viewer  2.0,  which  contains  a 
complete  range  of  media  features  and  web  functionality  this  obstacle  has  been,  for  the  most  part, 
averted.  One  appealing  aspect  of  the  Second  Life  platforms  is  the  communal  element,  which 
allows  researchers  to  solicit  input  and  showcase  the  experimental  environment  readily  with 
colleagues  in  education  and  government.  While  some  coding  work  is  still  necessary  to  complete 
integration  with  Viewer  2.0,  as  a  starting  point  for  designing  the  experimental  space,  Second  Life 
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offers  the  ability  to  quickly  construct,  edit,  and  deploy  assets  within  the  space  itself,  which  is 
crucial  in  the  early  design  phase.  See  screenshot  in  Figure  1. 


Figure  1  Second  Life  and  OpenSim 


Teleplace 

Teleplace  (fonnerly  Qwak)  offers  a  very  basic  virtual  environment  that  could  potentially  be  used 
for  ELICIT  experimentation.  While  it  contains  less  broad  functionality  and  naturalistic  elements 
that  are  native  to  the  Second  Life  platforms,  many  of  these  natural  elements  may  be  considered 
extraneous  to  the  ELICIT  experimental  design.  The  spaces  within  Teleplace  are  interior  only 
spaces,  with  no  access  to  natural  landscapes.  In  the  early  phase  of  the  project  development,  this 
is  not  an  issue.  However,  as  the  project  proceeds,  it  is  possible  that  researchers  may  want  to 
employ  exterior  spaces  as  part  of  the  research.  As  with  the  original  Second  Life  viewer, 
Teleplace  does  not  offer  any  JavaScript  functionality,  and,  unlike  the  OpenSim  aspect  of  Second 
Life,  Teleplace  does  not  provide  the  openness  of  code  to  all  this  addition.  That  said,  the  Python 
scripting  language  is  supported  by  Teleplace,  which  would  allow  developers  to  recreate  the 
scripted  content  of  ELICIT  within  this  platform.  However,  significant  development  time  would 
be  required  to  achieve  that  end.  Furthermore,  while  Teleplace  allows  for  customized 
environments,  these  modeled  structures  must  be  developed  outside  the  platform  using 
professional  modeling  tools,  therefore  eliminating  the  ability  for  rapid  prototyping  of  designs. 
The  designs,  that  are  imported,  however,  would  be  of  fine  quality.  The  difficulty  at  the  early 
phase  of  this  project  is  that  significant  reworking  of  these  models  would  be  required  to  make 
even  the  smallest  changes  to  the  experimental  spaces.  See  screenshot  in  Figure  2. 
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Figure  2  Teleplace 


O.L.I.V.E. 


OLIVE  (On-Line  Interactive  Virtual  Environment)  by  Forterra  is  one  of  the  most  versatile  and 
rigorous  of  the  various  virtual  world  platforms  examined.  Like  Second  Life,  OLIVE  is  a  broad 
environmental  platfonn,  with  a  native  naturalistic  environment  that  allows  for  both  interior  and 
exterior  spaces.  However,  like  Teleplace,  OLIVE  requires  that  modeled  structures  and  spaces  be 
created  outside  of  the  platform  and  imported.  As  such,  this  limits  the  ability  to  adapt  and  modify 
models  quickly  according  to  experimental  specifications.  The  functionality  of  OLIVE  allows  for 
all  the  basic  communication  and  person-to-person  interaction  as  in  both  Second  Life  and 
Teleplace.  One  key  advantage  that  OLIVE  offers  which  is  not  available  in  many  of  the  other 
platforms  is  JavaScript  support.  As  a  result,  researchers  could  seemingly  utilize  the  current  coded 
ELICIT  product  within  OLIVE  without  rewriting  or  adapting  the  coding  to  a  different  language. 
This  could  be  a  great  resource  saver,  in  that  time  and  money  would  need  not  be  invested  in 
recreating  the  ELICIT  functionality  within  the  virtual  environment.  Instead  researchers  would  be 
able  to  simply  bring  that  functionality  into  the  environment  itself.  While  not  suited  to  the  rapid 
prototyping  of  the  design  in  the  early  phases  of  the  project,  OLIVE  could  potentially  be  very 
useful  in  later  fully  immersive  ELICIT  experiments.  See  screenshot  in  Figure  3. 
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Figure  3  O.L.I.V.E. 


Protosphere 

Protosphere  is  one  of  the  most  widely  used  enterprise  level  virtual  world  platforms,  especially 
for  distributed  business  collaboration.  Like  Teleplace,  Protosphere  focuses  on  interior  meeting 
and  training  spaces,  rather  than  on  exterior  environments.  However,  unlike  Teleplace,  there  are 
exterior  spaces  that  can  be  viewed  through  windows,  creating  a  realistic  office  setting.  Like  the 
OLIVE  and  Teleplace  platforms,  specific  models  may  be  created  in  external  programs  and 
imported  to  customize  any  space.  However,  as  with  the  other  platforms  (except  Second 
Life/OpenSim),  this  restricts  the  speed  with  which  elements  may  be  changed  or  adapted.  Similar 
to  the  OLIVE  platfonn,  Protosphere  enables  the  use  and  sharing  of  any  document  or  application 
that  is  able  to  run  on  the  users’  desktop  environment  within  the  virtual  space.  The  ability  to 
natively  run  ELICIT  within  Protosphere  without  adapting  the  code  is  a  significant  benefit  of  the 
platform.  Functionally,  Protosphere  offers  all  of  the  communications  tools  required  to  use 
ELICIT  in  the  virtual  space.  Unlike  the  other  platforms  reviewed  which  are  all  able  to  run  on 
either  a  Windows  or  Macintosh  operating  system,  Protosphere  is  only  available  for  Windows  OS 
at  this  time.  See  screenshot  in  Figure  4. 
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Figure  4  Protosphere 


ELICIT  USE  CASE  DESIGN  IMPLICATIONS 

In  this  section,  we  draw  heavily  but  briefly  from  (Leweling  &  Nissen,  2007;  Powley  &  Nissen, 
2009)  to  summarize  the  current  ELICIT  interface  and  game  play.  This  helps  to  establish  a  use 
case  and  set  of  design  specifications  for  the  development  of  an  immersive  environment  for 
ELICIT :  succinctly,  we  wish  to  develop  a  virtual  environment  that  emulates  the  current  ELICIT 
environment  as  closely  as  possible  with  the  exception  of  people  playing  the  game  through 
avatars  instead  of  the  current  textual  interface.  We  begin  by  describing  this  task  environment  and 
then  outline  the  participants,  groups,  protocols,  controls,  manipulations  and  measurements  used 
typically  for  experimentation  via  ELICIT.  We  close  this  section  with  a  summary  of  design 
implications. 


Task  Environment 


The  task  requires  a  team  of  participants  performing  the  roles  of  intelligence  analysts  to 
collaborate  and  identify  a  fictitious  and  stylized  terrorist  plot.  The  fictitious  terrorist  plot  is 
described  through  a  set  of  informational  clues  called  “factoids”  that  have  been  developed 
systematically.  The  game’s  design  is  similar  to  the  Parker  Brothers’  board  game  “Clue”  in  that  it 
requires  each  player  to  analyze  clues  and  combine  assessments  with  other  players  to  identify  key 
aspects  of  the  fictitious  plot.  Each  factoid  describes  some  aspect  of  the  plot,  but  none  is  sufficient 
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to  answer  all  of  the  pertinent  questions  (i.e.,  Who  will  execute  the  attack?  What  is  the  target  to  be 
attacked?  Where  will  the  attack  take  place?  When  will  the  attack  take  place?). 

The  factoids  are  distributed  among  the  players  in  a  series  of  steps:  each  player  receives 
two  clues  initially,  followed  by  one  after  five  minutes  of  play  and  another  after  ten  minutes  have 
elapsed.  The  factoid  distribution  is  designed  so  that  no  single  player  can  solve  the  problem 
individually  and  that  the  team  of  players  cannot  solve  the  problem  until  after  the  final 
distribution.  In  other  words,  the  players  must  collaborate  to  solve  the  problem,  and  they  are 
required  to  do  so  for  a  minimum  of  ten  minutes.  Evidence  from  previous  experiments  (e.g., 
(Parity  Communications  Inc.,  2006)  suggests  that  play  requires  substantially  more  time  (e.g.,  an 
hour  or  more). 

Participants  play  the  game  in  one  or  two  modes.  1)  For  the  virtual  environment,  they  play 
via  ELICIT  client  applications  on  separate  computer  workstations  linked  to  a  Web  game  server. 
Each  subject  has  access  to  a  set  of  five  functions  supported  by  the  client  application:  1)  List,  2) 
Post,  3)  Pull,  4)  Share,  and  5)  Identify.  After  the  game  has  completed,  the  administrator  ends  the 
simulation  from  the  server  application.  The  ELICIT  application  captures  time-stamped 
interactions  (e.g.,  Pose,  Pull,  Identify,  List  functions)  including,  for  instance,  when  and  which 
factoids  are  distributed  to  each  player,  when  and  which  factoids  are  posted  to  which  common 
screens,  when  and  which  common  screens  are  viewed  by  each  player,  when  and  which  factoids 
are  shared  between  each  player,  and  the  time  stamped  results  of  each  player’s  Identify  attempt 
(i.e.,  to  identify  the  who,  what,  where  and  when).  2)  For  the  physical  environment,  they  play 
through  face-to-face  interaction  in  rooms  equipped  with  tables  and  white  boards.  Factoids  are 
time  stamped  and  distributed  to  the  players  on  pieces  of  paper,  and  paper  “postcards”  are  time 
stamped  and  used  to  collect  players’  Identify  attempts.  We  do  not  attempt  to  log  or  time  stamp 
players’  other  information  sharing  and  processing  activities. 

The  game  requires  considerable  cognitive  and  collaborative  effort  to  play  well  (i.e., 
identify  the  pertinent  details  of  a  terrorist  plot),  but  experience  indicates  that  such  effort  is  within 
the  capabilities  of  many  people  and  groups. 

Participants 

Participants  for  ELICIT  experiments  come  from  a  broad  variety  of  backgrounds  (e.g.,  ranging 
from  undergraduate  students  to  professional  intelligence  analysts).  In  most  studies,  participants 
are  organized  hierarchically  as  delineated  in  Figure  5.  Such  organization  stratifies  them  into  three 
functional  levels.  The  Senior  Leader  is  responsible  for  the  intelligence  organization  as  a  whole 
and  has  four  Team  Leaders  (middle  managers)  reporting  directly.  Each  team  leader  in  turn  has 
three  Team  Members  (Operators)  reporting  directly  and  is  responsible  for  one  set  of  details 
associated  with  the  terrorist  plot.  For  instance,  Team  Leader  (Who)  and  his  or  her  team  are 
responsible  for  the  “who”  details  (e.g.,  which  terrorist  organization  is  involved)  of  the  plot,  Team 
Leader  (What)  and  his  or  her  team  are  responsible  for  the  “what”  details  (e.g.,  what  the  likely 
target  is),  and  so  forth  for  “where”  and  “when.”  Participants  are  assigned  randomly  to  these  roles 
as  a  general  rule. 
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Figure  5  Hierarchical  Organization 


Treatment  Groups 

Participants  are  assigned  to  play  in  groups  of  17  members.  However,  many  experiments 
manipulate  the  organizational  structure  to  compare  efficacy  of  the  Hierarchy  (i.e.,  as  most 
intelligence  groups  are  organized  currently)  with  alternate  structures  (esp.  the  Edge  (Alberts  & 
Hayes,  2003).  As  suggested  above,  many  other  experiment  treatments  (e.g.,  alternate 
communication  technologies  (Thunhohn  et  ah,  2009),  knowledge  vs.  information  flows 
(Leweling  &  Nissen,  2007),  trust  (Powley  &  Nissen,  2009),  textual  vs.  physical  interactions 
(Wynn,  Ruddy,  &  Nissen,  2010),  others)  are  assessed  through  ELICIT  experiments,  and 
participants  are  assigned  generally  to  different  treatment  groups  according  to  each  specific 
experiment  design. 


Protocols.  Protocols  share  many  similarities  across  ELICIT  experiments.  Generally,  participants 
report  to  a  networked  classroom  on  their  assigned  day  for  the  experiment.  Once  seated, 
participants  are  allotted  ten  minutes  or  so  to  read  a  set  of  instructions  pertaining  to  both  the 
experiment  and  the  ELICIT  environment;  they  are  encouraged  to  ask  questions  about  the 
experimental  settings  and  environment.  Once  participants  read  the  instructions,  in  many  studies 
they  have  another  ten  minutes  or  so  to  discuss  their  approach  to  the  problem-solving  scenario 
with  others  in  their  group  and  take  a  short  break  before  beginning. 

Once  the  game  begins,  each  participant  receives  unique  factoids  in  three  phases:  1)  two 
factoids  initially  when  the  game  begins,  2)  one  after  five  minutes,  and  3)  one  at  the  ten-minute 
mark.  Role-specific  factoids  are  distributed  automatically  by  ELICIT  and  in  a  manner  ensuring: 
a)  that  no  player  can  solve  the  plot  alone,  and  b)  that  the  plot  cannot  be  solved  until  all  factoids 
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have  been  distributed.  Factoids  are  time  stamped  and  appear  automatically  on  players’  ELICIT 
screens,  after  which  they  can  be  posted,  pulled  and  shared.  As  noted  above,  all  Post,  Pull,  Share 
and  Identify  actions  are  time  stamped  and  logged  by  the  game  server. 

In  most  experiments,  participants  communicate  with  one  another  during  game  play  using 
only  the  computer-network  capabilities  supported  by  ELICIT  (esp.  Post,  Pull  and  Share);  no 
verbal  communication  is  allowed.  Although  some  prior  experiments  incorporate  chat  and  like 
technologies  as  manipulations  (Thunholm  et  ah,  2009),  such  technologies  are  not  pennitted  in 
most  study  protocols. 

Each  player  is  assigned,  randomly  in  most  studies,  a  pseudonym  associated  with  his  or 
her  role  in  the  game,  and  to  conceal  players’  identities  in  previous  experiments,  their 
pseudonyms  are  not  to  be  revealed.  This  proves  to  be  an  unenforceable  protocol  in  many  prior 
experiments,  however,  as  players  are  situated  often  in  the  same  room  and  able  to  ascertain  one 
another’s  pseudonyms. 

Additionally,  the  ELICIT  software  limits  participants’  Post  (i.e.,  sharing  factoids  with 
others)  and  Pull  (accessing  factoids  posted  by  others)  access  to  specific  common  screens  within 
the  Hierarchy  case.  Specifically,  those  players  in  the  “who”  group,  for  instance,  are  allowed  to 
Post  to  and  Pull  from  only  one  of  the  four  common  screens  (i.e.,  the  “who”  screen)  noted  above. 
The  only  exception  applies  to  the  Senior  Leader,  who  has  post-pull  access  to  all  four  common 
screens.  This  restriction  is  relaxed  in  most  experiments  examining  the  effects  of  alternate 
organizational  forms  (e.g.,  Edge). 

Also,  most  experiments  enable  every  player  to  share  factoids  with  any  other  player, 
regardless  of  team  assignment.  This  appears  to  conflict  with  the  post  and  pull  restrictions  from 
above  and  to  mitigate  the  effects  of  hierarchical  organization.  In  some  experiments  (Bergin  et  ah, 
2010),  however,  factoid  sharing  conforms  to  the  same  restrictions  set  up  for  post  and  pull  (e.g., 
within-group  only):  Operators  are  permitted  to  share  only  with  members  of  their  own  teams,  and 
Team  Leaders  are  permitted  to  share  only  with  Operators  on  their  respective  teams  in  addition  to 
the  Senior  Leader.  Further,  the  Senior  Leader  communicates  only  with  Team  Leaders.  This 
reinforces  hierarchical  communication  and  chain  of  command  and  provides  greater  contrast  with 
the  Edge  and  other  organizational  approaches. 

The  simulation  ends  after  approximately  45-60  minutes.  All  players  are  given  the  option 
to  Identify  the  plot  details  if  they  have  not  done  so  already.  In  many  experiments,  players  are 
instructed  to  Identify  only  once  during  game  play,  whereas  in  others  multiple  Identify  attempts 
are  permitted.  In  either  case  participants  are  incentivized  both  to  solve  the  plot  individually  as 
well  as  to  collaborate  so  that  others  on  the  team  (esp.  the  Senior  Leader)  solve  the  plot  quickly 
and  accurately. 

Measurements 

Measurements  of  dependent  variables  vary,  but  many  follow  variations  of  a  two-factor  the 
scheme  (Leweling  &  Nissen,  2007).  For  instance,  one  can  operationalize  performance  as  a  two- 
dimensional  dependent  variable  comprised  of:  1)  speed  (i.e.,  time  to  identify  plot  details 
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correctly)  and  2)  accuracy  (i.e.,  correct  identification  of  plot  details).  These  dependent  measures 
are  infonned  by  literature  in  the  psychological  and  organizational  domains  that  suggest  a  trade¬ 
off  exists  between  time  and  accuracy  in  tasks  requiring  high  cognition  and/or  advanced  motor 
skills  (e.g.,  see  (Beersma  et  al.,  2003;  Elliott,  Helsen,  &  Chua,  2001;  Guzzo  &  Dickson,  1996; 
Meyer,  Irwin,  Osman,  &  Kounios,  1998;  Plamondon  &  Alimi,  1997;  Rogers  &  Monsell,  1995)  at 
both  the  individual  and  team/group  levels  of  analysis. 

In  the  first  component,  speed  pertains  to  how  long  it  takes  a  participant  to  submit  his  or 
her  identification  of  the  terrorist  plot  details.  For  ease  of  comparison,  the  scale  for  this  speed 
measurement  is  normalized  often  to  a  0-1  scale,  with  1  being  more  desirable  (i.e.,  faster). 
Measuring  and  nonnalizing  time  is  straightforward,  as  the  time  for  each  participant’s 
identification  is  logged  to  the  nearest  second  by  the  software.  Specifically,  each  participant’s 
elapsed  time  is  recorded  when  he  or  she  uses  ELICIT  to  Identify  the  plot.  To  construct  a  scale  in 
which  faster  speeds  (i.e.,  shorter  times  to  Identify)  result  in  larger  values,  a  baseline  time  is 
established  as  the  maximum  time  required  for  the  slowest  of  all  participants  (i.e.,  3000  seconds 
in  this  experiment).  Each  participant’s  time  to  identify  is  related  to  this  baseline  and  nonnalized 
to  produce  a  scaled  score  according  to  the  fonnula:  speed  =  (3000  -  time)  /  3000;  that  is,  an 
individual  participant’s  time  (say,  for  example,  2375  seconds)  would  be  converted  to  a  speed 
score  as:  speed  =  (3000  -  2375)  /  3000  =  0.2083.  All  participants’  times  are  converted  to  speed 
scores  in  this  same  manner  and  using  this  same  baseline. 

The  second  component  of  performance,  accuracy,  refers  to  the  quality  of  the 
identification  of  the  impending  terrorist  attack  (i.e.,  Who,  What,  Where,  and  When).  Each 
participant’s  Identify  action  is  scored  with  a  value  of  1  for  each  correct  answer  to  the  Who,  What 
and  Where  aspect  of  the  solution.  Note,  however,  that  the  When  aspect  of  the  solution  includes 
three  components  (i.e.,  Month,  Day,  and  Time).  In  order  to  avoid  weighting  this  aspect  more 
heavily  than  the  other  three,  each  participant’s  Identify  action  is  scored  with  a  value  of  1/3  for 
each  correct  answer.  The  resulting  sum  is  divided  by  four  to  construct  a  [0-1]  scale;  that  is,  an 
individual  participant’s  Identify  (say,  for  example,  identifies  the  Who,  What  and  Where  aspects 
correctly  but  is  correct  only  on  the  day  and  not  the  month  or  time  components  of  the  When 
aspect)  would  be  converted  to  an  accuracy  score  as:  accuracy  =  (1  +  1  +  1  +  1/3)  14  =  0.83. 

Design  Implications 

With  this  description  we  have  the  basis  of  a  use  case  for  ELICIT.  The  central  design  task 
requires  preserving  as  much  of  the  typical  ELICIT  play  as  possible  while  exploring  the 
implications  of  such  play  being  performed  in  immersive  virtual  environments.  In  terms  of  virtual 
environments,  most  of  the  previous  research  has  focused  on  the  experimental  response  to  human 
representation  (avatar)  and  its  activity,  and  not  on  the  environment  itself.  This  research  aims  to 
understand  what  impact,  if  any,  the  environment  specifically  has  on  participants  that  could 
potentially  improve  perfonnance  within  ELICIT. 

Currently  ELICIT  is  a  solitary  activity,  with  participants  separated  in  a  manner  that  is 
analogous  with  real  world  intelligence  sharing.  In  designing  an  ELICIT  experience  where 
participants  may  have  access  to  one  another,  share  a  common  space,  or  discuss  various  factoids, 


12 


this  research  seeks  to  understand  how  performance  may  be  enhanced  by  this  collective 
environment. 

By  adapting  ELICIT  for  play  within  a  virtual  environment,  two  new  elements  of  the 
experience  are  being  added: 

1)  The  ELICIT  game -play  will  not  take  place  in  an  environment  as  a  discernable  place. 

2)  Participants  experience  themselves  as  present  within  that  space,  and  co-present  with  the 
other  participants. 

This  research  seeks  to  understand  how  the  addition  of  a  spatial  dimension  impacts  the 
performance  of  participants  within  the  game,  and  whether  it  supports  more  successful  game-play, 
or  whether  it  detracts  from  the  success  of  participants. 

PRELIMINARY  VIRTUAL  ENVIRONMENT  DESIGN  AND  PROTOTYPE 

Environments  are  by  definition  complex.  They  envelope  and  surround  us,  and  even  the  most 
simplistic  environment  is  filled  with  a  staggering  range  of  elements  and  stimuli.  This  is  true 
whether  that  environment  is  physical  or  virtual,  as  they  both  contain  aspects  that  have  varying 
types  of  emotional  valence.  In  order  to  determine  the  affective  charge  of  each  environmental 
element  in  the  adaptation  of  ELICIT,  it  is  important  to  begin  this  experimental  process  with  a 
neutrally  toned  environment  from  which  to  build  on. 

The  first  step  in  this  process  is  to  limit  extraneous  elements  from  the  preliminary  design. 
To  that  end,  designers  are  utilizing  the  International  Affective  Picture  System  (IAPS)  in  order  to 
understand  the  emotional  valences  of  various  images,  and  to  use  that  catalogue  of  affects  to 
determine  a  properly  neutral  environmental  design  for  the  ELICIT  experimentation  to  take  place 
in. 


The  IAPS  images  are  a  series  of  emotionally  charged  representations  that  are  used  in 
various  types  of  experiments  to  add  a  pre-determined  affective  tone  to  the  content  (Lang, 
Bradley,  &  Cuthbert,  2008).  Researchers  already  understand  the  potency  of  this  imagery,  noting 
that  a  single  valence  determined  image  can  “put  someone  in  an  emotional  state  lasting  several 
minutes”  (Wilson,  2005). 

While  three-dimensional  virtual  environments  seem  like  a  distant  cousin  to  static 
photographic  images,  it  should  be  noted  that  virtual  platforms  that  are  deemed  3D  are  in  fact  2D 
environments  where  the  illusion  of  three-dimensions  is  created  by  placing  the  participant  in  a 
surrounded  landscape  and  then  allowing  the  participant  to  navigate  this  all  360  degrees  of  this 
space.  This  effectively  creates  3D  space,  but  only  as  perceived  by  the  user.  In  fact,  the  space  is 
projected  on  a  flat  panel  computer  monitor,  and  is  no  more  3D  than  the  current  ELICIT  game- 
play.  However,  from  an  experiential  point  of  view,  this  forced  3D  perspective  is  enough  to  give 
the  participant  the  sense  that  they  are  operating  in  a  realistic  space.  See  screenshot  in  Figure  6. 
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Figure  6  Neutral  Space  Prototype 


There  is  an  interesting  and  unexplored  analogy  between  IAPS  affectively  charged 
imagery  and  its  impact  on  behavior,  and  the  visual  stimuli  of  virtual  environments.  Participants 
in  these  worlds  are  not  simply  being  displayed  or  flashed  an  affectively  biased  image,  but  are  in 
fact  deeply  immersed  and  emotionally  involved  with  the  environment  itself.  That  environment  is 
constructed  by  a  series  of  two-dimensional  textures  to  create  the  optical  illusion  of  three- 
dimensional  space.  Therefore  it  is  believe  that  a  careful  study  of  the  affective  quality  of  the  IAPS 
imagery  will  lead  designers  to  understand  how  to  develop  virtual  environments  with  a  specific 
emotional  tonus. 

This  stimuli  classification  system  provides  designers  with  a  range  of  imagery  that  falls 
into  broad  categories  like  positive  stimuli,  negative  stimuli,  and  neutral  stimuli.  By  analyzing  the 
imagistic  components  of  IAPS,  virtual  environment  designers  are  able  to  include  aspects  and 
elements  of  the  design  to  achieve  a  neutral  environment  for  experimentation.  Furthermore,  this 
design  process  will  help  develop  a  “pattern  language”  of  virtual  environment  design.  “A  pattern 
is  an  instruction,  which  shows  how  this  spatial  configuration  can  be  used,  over  and  over  again,  to 
resolve  the  given  system  of  forces,  wherever  the  context  makes  it  relevant"  (Alexander,  1977). 

Out  of  this  work,  a  lexicon  of  imagery  and  an  inventory  of  archetypal  stimuli  will  emerge 
that  can  be  employed  in  future  virtual  environment  designs  to  facilitate  various  experimental 
parameters  as  well  as  influence  the  experimentation  with  specific  emotional  valences. 
Researchers  in  the  present  study  of  ELICIT  game-play  will  understand  which  kinds  of 
environmental  stimuli  in  the  virtual  environment  support  enhanced  performance  and  which 
detract  from  it. 


14 


Aside  from  the  affective  components  of  the  design  that  an  analysis  of  IAPS  will  reveal,  there 
are  certain  obvious  elements  that  are  built  into  the  initial  design  of  the  ELICIT  game  space  that 
will  support  its  neutral  valence.  These  aspects  include: 

1)  Creation  of  interior  only  spaces  free  from  the  complexity  of  the  natural  world  including 
sunlight,  plants,  landscape  variances  (water  to  land),  clouds,  and  all  other  natural 
elements  that  do  not  support  the  experimental  objectives. 

2)  Subtle  texturing  of  spaces  so  as  not  to  overtly  determine  building  materials.  As  texture  is 
one  aspect  of  imagery  that  impacts  emotional  response,  designers  need  to  be  aware  not  to 
overly  texture  the  spaces.  Therefore  building  elements  that  reduce  specific  reference  to 
their  materials,  for  example  dry  wall  rather  than  bricks,  linoleum  flooring  rather  than  hard 
wood,  and  tightly  woven  fabric  textures  rather  than  broad  weaves,  need  to  be  employed 
to  reduce  emotional  stimuli. 

3)  Constant  ambient  lighting  will  also  create  a  comfortable  space  for  visual  data  without 
dramatic  inputs  from  spot  lighting  or  the  natural  light  occurring  through  windows. 

4)  Narrow  range  of  inputs  and  the  exclusion  of  distracting  elements  will  help  the 
environment  focus  participants  on  the  task  oriented  experimentation,  rather  than  on  the 
environmental  elements  contained  therein. 


Figure  7  Realistic  Texturing  Changes  Emotional  Tone 
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While  this  environment  will  not  be  a  distinguishable  real  place,  it  is  a  place  nonetheless.  It  is 
intended  that  the  experimental  virtual  environment  will  feel  like  a  real  place  to  the  participants, 
while  not  carrying  any  overtly  determined  emotional  charge.  Participants  need  feel  they  are 
“somewhere,”  that  place  must  function  to  support  game-play,  and  the  participants  must  also 
experience  co-presence  in  that  space  with  the  other  participants.  See  screenshot  in  Figure  7. 

The  “somewhere”  that  researchers  have  decided  on  as  a  starting  environment  for  experimentation 
is  based  on  a  physical  space  previously  utilized  for  ELICIT  experiments.  While  not  a  specific 
place  per  se,  this  hybrid  environment,  based  on  a  real  physical  room,  connects  in  a  meaningful 
way  to  the  previous  ELICIT  runs.  This  hybrid  space  follows  the  criteria  outlined  above  for  the 
specific  qualities  of  a  neutral  space.  However,  it  is  believed  that  because  this  hybrid  environment 
is  closely  related  to  rooms  used  in  previous  experiments,  it  will  allow  researchers  a  more  precise 
point  of  comparison  when  analyzing  results  from  the  virtual  space  and  from  the  physical  lab.  See 
screenshot  in  Figure  8. 


Figure  8  Hybrid  room  based  on  physical  experimentation  space 


CONCLUSION 

Research  in  command  and  control  is  advancing  rapidly  through  a  campaign  of  laboratory 
experimentation  using  the  ELICIT  (Experimental  Laboratory  for  Investigating  Collaboration, 
Information-sharing,  and  Trust)  multiplayer  online  counterterrorism  intelligence  game.  In  most 
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ELICIT  experiments,  participants  play  the  game  through  a  Web  interface  and  interact  with  one 
another  solely  through  textual  information  exchange.  This  mirrors  in  large  part  the  network¬ 
centric  environment  associated  with  most  counterterrorism  intelligence  work  in  practice. 
However,  we  argue  that  a  more  immersive  virtual  environment  offers  potential  to  improve 
performance.  The  research  described  in  this  article  addresses  the  preliminary  design  and 
prototyping  of  an  immersive  and  dynamic,  virtual  environment  for  ELICIT  experimentation,  in 
which  participants  play  the  game  through  avatars  representing  them.  We  summarize  a  number  of 
alternate  virtual  environment  platforms  available  to  researchers  for  work  such  as  this,  and  we 
explain  the  rationale  for  the  specific  environment  selected  for  this  project.  Additionally,  we 
describe  the  prototypical  design  strategies  for  adapting  game  play  into  virtual  environments  and 
characterize  preliminary  case  examples  to  understand  the  differences  and  potential  benefits  of 
adapting  ELICIT  game  play.  This  research  lays  a  foundation  for  moving  experimentation  into 
immersive  virtual  environments. 
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